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BACKGP-C'JT'.T) 


In  response  to  inquiries  from  Congressional  representa¬ 
tives,  the  Acting  Assistant  Secretary  of  Defense  (Health 
Affairs)  requested  that  the  Army  document  a  Department  of 
Defense  (DoD)  position  regarding  an  extension  of  the  Fort  Bragg 
Mental  Health  Demonstration  Project.  It  was  requested  that  the 
Army  establish  a  panel  of  Army/DcD  experts  (psychiatrists, 
psychologists,  other  clinicians,  and  clinical  statisticians)  to 
review  the  evaluation  and  other  related  data  concerning  the 
Demonstration  Project  in  order  to:  (l)  support  a  DoD  position 
on  the  level  of  confidence  necessary  to  confirm  treatment 
results/conclusions,  and  (2)  indicate  the  impact  of  an  Army 
approved  evaluation  due  date  on  that  level  of  confidence. 

This  technical  report  presents  an  independent  statistical 
analysis/review.  No  actual  data  from  the  Fort  Bragg 
Child/Adolescent  Mental  Health  Demonstration  Project  or  the  Fort 
Bragg  Evaluation  Project  were  made  available.  However, 
information  contained  in  a  letter  .shown  as  Appendix  A)  written 
by  Dr.  Lenore  Behar,  Ph.D.,  Head  of  the  Child  and  Family 
Services  Branch,  North  Carolina  Department  of  Human  Resources, 
to  Mr.  Leo  Sleight,  Central  Contracting  Office,  Department  of 
the  Army,  Headquarters  U.S.  Army  Health  Services  Command,  Fort 
Sam  Houston,  Texas,  dated  February  15,  1993,  was  provided  by 
Vanderbilt  University.  In  the  letter.  Dr.  Behar  presented  two 
data  collection  plans.  These  plans,  one  Short-Term  and  one 
Long-Term,  differ  in  the  number  of  cases  collected  at  'Wave  3'. 
The  effectiveness  of  each  plan  was  described  by  means  of  a  power 
value  of  a  statistical  test  for  detecting  differences  in 
improvement  in  mental  health  outcomes  between  Demonstration  and 
Comparison  cases.  In  addition,  a  reprint  of  a  paper  submitted 
to  the  1992  American  Psychological  Association  Convention 
addressing  power  analysis  in  psychotherapy  research  was 
furnished.  This  paper  is  included  as  Appendix  B.'  Also 
submitted  was  documentation  supporting  the  power  values  in 
Appendix  A  in  materials  attached  to  a  letter  dated  April  30, 
1993,  written  by  Dr.  Leonard  Bickman,  Ph.D.,  Director  of  the 
Center  for  Mental  Health  Policy,  Institute  for  Public  Policy 
Studies,  Vanderbilt  University,  to  LTC  Thomas  E.  Leonard, 
Headquarters  U.S.  Army  Health  Services  Command,  Fort  Sam 
Houston,  Texas.  Pertinent  portions  of  this  documentation  are 
included  as  Appendix  C. 


POWER  ANALYSIS  COMPARISON 
OF  TWO  DATA  COLLECTION  PLANS 


Power  Analysis  Assumptions. 

In  the  statistical  assumptions  presented  in  Appendix  A, 
the  type  of  variable (s)  used  to  measure  'improvement'  between  an 
average  Demonstration  case  and  an  average  Comparison  case  was 
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not  defined.  The  paper  shown  in  Appendix  5  was  referenced 
instead,  presenting  tr.e  results  of  a  meta  -  analysis  f  or  12 
categories  of  outcome  measures,  six  each  for  behavioral  and 
nonbehavicral  treatments.  It  appears  that  the  Fort  Bragg 
Evaluation  Project  used  the  Appendix  E  paper  to  obtain  the  ''aiue 
of  the  effect  size  (ES)  for  Normed  Racing  Scales- -Nonbehavioral 
Treatment  outcome  measures- -as  this  value  is  included  in 
Appendix  A.  In  Appendix  A  (p.  A-6} ,  it  is  stated  that  the 
Short-Term.  Plan  has  50%  power  and  the  Long-Term  Plan  of  data 
collection  would  have  30%  power.  These  levels  of  power  were 
based  on  a  simulation  m.odel  submitted  by  Vanderbilt  University 
(Appendix  C) . 

The  effect  size  (ES)  index  identified  as  d  by  Cohen 
(1988),*  is  Che  standardized  difference  between  two  population 
means.  This  equation  is  as  follows: 


o 

where  d  =  ES  index  for  c  test  of  means, 

=  population  means, 

and  cr  =  standard  deviation  of  either  population 
(equal  variance  is  assumed)  . 


The  effect  size  value  (SS  *  0.25)  derived  in  Appendix  B  (p.  B-2) 
and  cited  in  Appendix  A  (p.  A- 5)  should  be  used  with  caution  for 
several  reasons.  First,  this  value  was  computed  for  a  series  of 
12  sub-group  samples.  The  Normed  Rating  Scale  used  to  derive 
the  power  in  Appendix  A  was  based  on  a  mean  sample  of  only  33 
cases.  The  authors  of  the  Appendix  3  paper  stated  this  problem 
of  variability  as  follows  (p.  3-2):  "The  large  discrepancies 
between  sample  sizes  actually  used  and  those  necessary  to  attain 
an  acceptable  level  of  power  in  the  studies  shown  in  Table  1 
make  it  difficult  to  assess  how  closely  the  obtained  treatment 
effect  sizes  represent  true  population  effects.  This,  in  turn 
underscores  Che  need  for  researchers  to  attend  to  power 
considerations  when  planning  therapy  outcome  studies."  When  a 
meta-analysis  is  based  on  such  a  small  size  the  probability  of 
error  is  high.  As  a  result,  the  mean  effect  size  (ES  =  0.25) 
used  in  Appendix  A  may  or  may  not  express  score  distances  (in 
units  of  variability)  for  the  actual  variables  measuring  health 
outcome  in  the  Fort  Bragg  Evaluation  Project. 

Secondly,  there  is  always  a  risk  that  meta-analysis  may 
have  employed  inappropriate  assumptions  with  regard  to  the 
validity  of  pooling  and  generality.  For  instance,  the  meta¬ 
analysis  may  contain  some  bias  as  to  how  the  outcome  should  be 
produced,  excluding  some  relevant  trials  from,  analysis.  In 
ocher  instances,  meta-analysis  may  use  multiple  results  from  the 
same  study,  and  because  the  results  are  not  independent  they  may 


bias  or  invalidate  the  meta -analysis .  In  other  cases,  the 
independent  studies  may  include  different  measuring  techniques 
and  definitions  of  variables,  so  the  outcomes  may  not  be 
comparable.  In  general,  effect  sizes  in  unique  areas  are  likely 
to  be  small  (ES  =  0.20  or  ES  =  0.30),  but  only  a  pilot  test 
would  give  an  answer  as  to  the  probable  magnitude  of  the  ES 
index  for  the  particular  variable  of  interest  in  a  particular 
situation. 

The  power  and  sample  size  tables  (Cohen,  1988)^  for  the 
above  specified  ES  =  0.25  in  Appendix  A  are  designed  to  analyze 
the  difference  between  means  of  two  independent  sam.ples  of  the 
same  size  drawn  from  normal  populations  with  equal  variances 
(using  the  t  test  for  means) .  If  these  assumptions  cannot  be 
made,  which  is  often  the  case,  the  additional  adjustments  that 
follow  are  explicitly  supported  by  Cohen  il98H)^  and  others. 
Computations  should  be  performed  to  obtain  the  hanr.onic  mean  if 
samples  of  different  sizes  but  equal  variance  are  present,  and 
the  root  mean  square  should  be  computed  if  two  samples  cf  the 
same  size  .having  unequal  variances  are  present.  If  both  sample 
sizes  and  variances  differ,  the  values  for  power  formulas  fro.m 
the  tables  cited  in  Appendix  A  may  not  be  valid. 

Since  no  actual  data  were  available  from  the  Fort  Bragg 
Evaluation  Project,  this  review  will  utilize  the  data  used  by 
Vanderbilt  University  for  this  analysis.  Appendix  A  contains  a 
comparison  of  the  two  data  collection  plans  using  power 
analysis.  The  Appendix  A  power  analysis  comparison  presents  the 
number  of  cases  after  attrition  for  both  the  Short-Term  and 
Long-Term  Plans  (p.  A-6) .  For  the  Short-Term  Plan,  299 
Demonstration  cases  and  150  Comparison  cases  were  expected.  The 
following  power  analysis  is  based  on  Cohen's  formulas  and  uses 
the  information  supplied  in  Appendix  A.  This  analysis  is 
followed  by  a  discussion  of  the  simulation  submitted  by 
Vanderbilt  University  and  included  as  Appendix  C. 


Power  Analysis  of  Short  and  Long-Term  Plans. 

Under  the  assumption  that  the  variances  in  the 
Demonstration  and  Comparison  sites  are  equal,  the  harmonic  mean 
(n)  of  the  Demonstration  sample  size  (.n^)  and  the  Comparison 
sample  size  (n^)  is  given  by  the  formula  (Cohen,  1988) 

n  =  ^  2  (299)  (150)  ^  89,700  ^  200 

rir  299  +  150  ~  449 


The  value  for  power  of  the  t  test  of  the  Demonstration  case  mean 
(mj))  and  the  Comparison  case  mean  (m^-)  testing  the  null 
hypothesis  t.hat  =  m<-  at  oc,  =  0 . 05  (one-tailed  test;  (Table 
2.3.2  from  Cohen,  1988)'^  gives  the  following  results: 
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for  n  =  200  and  ES  =  0.20,  power  =  0.64,  and 
for  n  =  200  and  ES  =  0.30,  power  =  0.91. 

The  effect  size,  proposed  in  Appendix  A  and  derived  from  a  meta- 
analysis  performed  in  Appendix  B,  is  0.25.  A  linear 
interpolation  was  performed  to  derive  the  power  of  the  t  test 
for  ES  ■  0.25.  This  computation  yielded  a  power  of  0.78  for  ES 
-  0.25,  ocj  SI  0.C5  and  n  =  200.  This  power  of  0.78  (78%),  as 
computed  for  the  Short-Term  Plan,  is  much  higher  than  the  0.50 
(50%)  quoted  in  Appendix  A.  A  full  precision  computation  of  the 
power  for  the  Short  and  Long-Term  Plans  is  presented  in  the  next 
section  of  this  report. 

The  Long-Term  Plan  projects  426  Demonstration  cases  and 
361  Comparison  cases.  This  harmonic  mean,  computed  under  the 
assumption  that  the  variances  are  the  same,  is  as  follows 
(Cohen,  1988) 

^  ^  2  (426)  (361)  ^  307,57  2  ^  g  ^ 

426  +  361  7  87 


Employing  Taible  2.3.2  in  Cohen  (1988),*  n  «  350  yields  power  = 
84%  for  ES  =  0.20  and  power  =  99%  for  ES  *•  0.30.  For  n  =  400, 
power  =  88%  for  ES  =  0.20  and  power  is  greater  than  99%  for  ES  = 
0.30.  The  linear  approximation  yields  a  power  of  93.3%  for  ES  •= 
0.25  (for  n  =  391)  . 


Computational  Procedure  for  the  Exact  Power 
of  the  Short  and  Long-Term  Plans. 

The  linear  interpolation  to  compute  power,  discussed  on 
pages  3  and  4,  was  justified  by  its  simplicity  and  by  the 
relatively  accurate  values  obtained.  The  full  precision  in 
computing  the  power  for  the  Short  and  Long-Term  Plans  was  based 
on  the  expression  (Cohen,  1988) 


_  _ d(n-l)  y/2n _ 

"  2(n-l)  1.21  -  r.06) 


where 


d 

and  n 


the  percentile  of  the  standard  normal 
distribution  giving  the  power  value 
the,  percentile  of  the  standard  normal 
distribution  for  otj  significance  level 
the  effect  size  ES 
the  harmonic  mean. 
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For  Che  Short -Tem  Piar*,  the  following  information  was 
available : 


n 

d 


200 

0.05 

0.25 

1.645. 


The  Zj.g  percentile  was  computed  under  these  assumptions  from  t.he 
aJbove  formula: 

=  (0.25)  (200  -  l)v/2l200)  _  . 

2(200  -  1)  +  1.21  (1.645  -  1.06) 

^  (0.25)  (199)  (20)  _  1  645  =  ^^5  _ 

398  +  (1.21)  ;0. 585)  ‘  398.708 

=  2.496  -  1.645  =  0.851. 


The  probability  for  this  percentile  was  found  from  the 
Normal  Curve  Areas  Table  C  (Daniel,  1988)  This  probability 
presents  the  power  of  the  test  and  is  equal  to  80.258%.  The 
Short-Term  Plan  gives  a  statistical  power  (computed  w?th  full 
precision)  exceeding  80%. 

A  similar  computation  was  performed  for  the  Long-Term  Plan 
under  the  following  assicmpticns : 

n  =  391 
a,  =  0.05 
d  =  0.25 
z,^,  =  1.645. 

The  2,.^  percentile  found  from  the  same  formula  (Cohen,  1988)“ 
was  computed  as  follows: 

_  =  (0.25)  (391  -  l)v^(2)  (391)  _  ^ 

2(391  -  1)  +  1.21  (1.645  -  1.06) 

=  (9*7  -5)  (27 .964)  _  .  ^  2,726 . 516  _  ^ 

780  +  0.70785  ■'  ~  780.708  ‘ 

=  3.492  -  1.645  =  1 . 847 . 

The  power  for  this  value  of  z,.j  found  from  the  Normal  Curve 
Areas  Table  C  (Daniel,  1988)'*  is  equal  to  96.78%. 


Additional  Power  Corp.putaticns . 


The  power  analysis  shown  aJoove  projects  that  the  number  of 
cases  in  the  Short-Term  Plan  is  currently  sufficient  to  draw 
statistically  significant  conclusions  with  high  statistical 
power  (80.258%).  An  additional  reason  for  this  conclusion  is 
found  by  using  the  sample  size  tadbles  provided  by  Cohen  (1988)'^ 
and  deriving  the  sample  size  necessary  to  achieve  full  80% 
power.  Sample  size  tables  provide  data  for  two  homogeneous 
normally  distributed  populations  from  which  random  samples  of 
the  same  size  were  derived.  The  ES  specified  in  Appendix  A  is 
0.25.  This  ES  level  is  not  taibulated  by  Cohen  (1988).'^ 
Therefore,  to  find  the  sample  size  for  an  untabulated  effect 
size,  the  following  formula  is  used  (Cohen,  1988) 


n 


n 


.  10 


100^2 


1 


where  n,Q  is  the  sample  size  for  desired  power, 
given  <x  and  ES  =  0.10, 
and  d  is  the  effect  jize. 


In  addition,  if  the  sample  sizes  are  not  equal,  one  sample  size 
is  treated  as  if  fixed,  while  the  other  is  computed,  vruen  the 
choice  is  arbitrary,  it  is  generally  supported  that  nc  be  fixed 
and  no  be  computed.  To  find  np,  the  following  formula  is  used 
(Cohen,  1988):'* 


2n^  -  n 


where 

n 

and  n^ 


fixed  sample  size  (Comparison  sites), 
value  read  from  the  Table  2.4.1  (Cohen, 

1988)^^  or  computed  from  the  previous  equation, 
sample  size  for  the  Demonstration  site. 


With  the  objective  to  determine  the  Demonstration  case 
sample  size  required  to  yield  a  power  =  80%  with  oc,  =  0.05  and 
ES  =  0.25,  and  fixing  the  Comparison  cases  at  n  =  150  (the 
current  level),  the  formula  for  computing  n  is: 


n 


■  +  1  =  - IslH -  +  1  =  ^^7  +  1  =  198  +  1  =  199. 

100d2  100(0.25)^  6.25 


'Source:  Table  2.4.1  (Cohen,  1988}.'^ 
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Next,  ^  value  is  put  into  the  formula  for  n,,: 


_  (150)  (199)  _  29 ,850 

~  2n^  -  n  2(150)  -  199  300  -  199 


29 , 850 
101 


295.54 


296  . 


Consequently,  296  Demonstration  site  patients  are  needed  to 
assure  an  80%  power  for  the  test  investigating  the  difference  in 
mental  health  outcomes  between  Demonstration  and  Comparison 
patients  (299  were  projected  in  Appendix  A)  . 

The  identical  procedure  was  applied  to  the  Long-Tenn  Plan. 
Given  that  the  Comparison  sites  consist  of  361  cases,  and 
assuming  the  same  conditions  (oc,  =  0.05,  ES  =  0.25,  power  = 

0.80),  a  sample  size  of  138  cases  for  the  Demonstration  site  was 
obtained : 


n  =  — +  1  =  199 
lOOd^ 

.  (361)  (199)  _  71,839  _  71,839 

“  2nc  -  n  ~  2  (361)  -  199  "  722  -  199  '  523 

=  1-7.36  =  138. 


As  proposed,  in  Appendix  A,  the  Long-Term  Plan  is  projected  to 
produce  426  Demonstration  cases.  Using  Vanderbilt  University's 
information  taken  from  Appendix  A,  the  above  analysis  computes 
only  138  cases  are  statistically  necessary  to  achieve  80%  power 


Assessment  of  the  Simulation  Method.  = 

Vanderbilt  University's  use  of  the  Monte  Carlo  simulation 
method  to  perform  a  power  analysis  in  the  present  situation  is 
an  inappropriate  application  of  this  type  of  simulation.  Using 
simulation  to  compute  the  power  analysis  without  any  information 
about  the  actual  data  is  not  an  appropriate  use  of  either 
simulation  or  power  analysis.  Concerning  simulation.  Miller  and 
Starr  (1969)’®  state; 

".  .  .Simulation  is  not  a  substitute  for  knowledge 
[emphasis  by  authors]  .  This  cannot  -t.-  over¬ 
emphasized.  Simulation  is  not  a  method,  which, 
somehow,  compensates  for  lack  of  knowledge." 
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In  general,  sir’.uiacion  should  re  treated  as  a  1600.01™:'?  ol  ''las 
resort"  (Naylor,  1971), to  be  used  only  when  analyrical 
techniques  are  net  available  for  obtaining  solutions  to  a  given 
model.  Power  analysis  gives  the  correct  probability  of  gettin 
a  significant  result  of  Comparison  and  Demonstration  site  mean 
only^when  the  effect  site  is  computed  precisely  (i.e.,  based  on 
actual  data  from  actual  variables  in  the  experiment  under 
consideraticn) . 

The  use  of  simulation  requires  complete  information  about 
the  process  or  object.  In  order  to  si.mulate  reasonably,  the 
probability  distributions  of  the  variables  of  interest  should  be 
known.  If"  these  distributions  are  not  known,  it  is  im^possible 
to  sim.ulate  the  process.  This  position  is  strongly  emphasized 
by  many  authorities  in  operations  research  (Naylor;  Ignizio  and 
Gupta;" Buf fa:  Smith;  Banks  and  Carson;  Gibra;  and  Miller  and 
Starr).-'  It  is  critical  t.hat  estimates  of  parameters  of  the 
sim.ulaticn  ."odel  be  derived  on  the  basis  of  observations  taken 
from,  the  actual  data.  Naylor  .1971)"  states: 

"...  There  is  ver-y  little  to  ba  gained  by  using  an 

inadeo-uata  r.cdel  to  carri/  cut  simulation  experiments 

on  a  ctmputer  because  we  would  merely  be  simulating 

cur  own  ignorance .  " 

Since  the  Monte  Carlo  technique  presented  in  Appendix  C  does  not 
involve  actual  data,  the  results  obtained  from  this  method  may 
be  entirely  m.isleading  and  not  accurate.  The  simulation  shown 
in  Appendix  I  is  based  on  assum.ptions  regarding  the  effect  size 
(ES  =  0.25).  This  value,  derived  from,  meta- analysis  (Appendix 
B,  p.  5-2) ,  may  not  apply  to  real  differences  between  the  mean 
values  of  .mental  health  outcom.es  for  the  Demonstration  and 
Com.pariscn  sites.  Another  assum.ption  (Appendix  A,  p.  A-5), 
regarding  the  average  child  im.provem.ent  by  0.3  SD,  due  to 
treat.ment  and  tim.e,  is  only  theoretical  because  it  is  .not  based 
on  aCtiucL.!.  dct-ci. 

As  stated  abcv e ,  Monte  Carlo  simulation  should  only  be 
utilized  when  direct  data  analysis  cannot  be  performed  (Gibra, 
1973),-'  which  IS  not  the  case  with  the  Fort  Bragg  Evaluation 
Project.  In  addition,  the  real  probability  distributions  of  all 
the  random  variables  under  consideration  m.ust  be  given  (Gibra, 
1973),-'  a  fact  ig.nored  in  Appe.ndix  C.  T.he  Monte  Carlo  m.ethod 
gives  only  approxim.ations  to  sampling  distributions  (Snedecor 
and  Cochran,  1980). ■'  To  this  extent,  the  technique  itself  is 
subject  to  sam.pling  error. 

Another  obsem/ation  about  the  Appendix  C  discussion  was 
chat  the  Monte  Carlo  method  was  performed  only  for  one  variable 
(CBCLj;  no  ether  variables  were  used.  The  analysis  might  had 
different  results  if  the  other  variables  were  considered. 

Finally,  beftre  any  simulation  m.odel  can  be  accepted  it  must  be 
verified  and  -.-alidated  to  identify  model  biases  and  erroneous 
assum.ptions,  if  any.  The  authors  of  the  m.odel ing  as  reported  in 
Appendix  C  ir.tiuded  no  such  validation. 


O'*  ff) 


Withou'  the  tse  cf  actuel  data,  the  effect  size  value 
(derived  frcn  the  r.eta -analyses  cited  in  Appendix  B}  was  usea  to 
calculate  the  power  in  this  report.  This  effect  size  was 
recommended  by  the  staff  of  t.-.e  Fort  Bragg  Evaluation  Project. 
Although  not  considered  actual  data,  the  effect  size  allowed  for 
no  additional  bias  to  be  created  by  the  Monte  Carlo  method.  The 
equations  used  to  com.pute  the  power  of  the  test  of  means  in  this 
report  are  supoorted  by  numerous  authorities  in  power  analysis 
(Cohen,  1988 


CC.TCLUSIC.N' 


The  power  values  for  the  directional  tests  computed  in 
this  study  and  the  values  given  in  the  proposal  in  Appendix  A 
are  signif icantly  differe.nt.  Utilizing  information  available  in 
Appendix  A  and  a  methodology  well  supported  in  the  statistical 
literature,  this  study  demonstrates  that  the  Short-Term  Plan 
would  yie^d  power  exceeding  81%  (80.258%)  at  full  precision, 
instead  of  51%  as  presented  in  Appendix  A.  Even  using  linear 
interpolation,  a  power  of  78%  was  derived.  This  study 
demonstrates  that  it  is  unnecessary  to  extend  the  duration  of 
the  project  based  cn  power  reguirem.ents ;  the  Short-Term  Plan 
should  produce  high  power  to  demonstrate  significance  if  the 
alternative  hypothesis  is  true.  The  Demonstration  sample  size 
n^  needed  to  achieve  80%  power  for  the  Short-Term  Plan  (oc  = 

0.05,  He  =  150,  ES  =  0.25)  ep-als  296  cases. 

Secondly,  because  the  standardized  effect  size  is  a 
computed  variable,  it  can  be  .modified.  This  modification  can  be 
achieved  by  any  of  several  m;ethods  currently  available  to  the 
Fort  Bragg  Evaluation  Project  staff  without  any  project 
extension.  Variance  can  be  reduced,  thereby  allowing  a  decrease 
in  sample  size  necessary  to  detect  a  particular  level  of  effect 
size  at  a  specified  power  by  increasing  quality  control  in  data 
collection  and  preparation  fer  analysis.  For  example,  each 
outccm.e  should  be  used  in  as  sensitive  a  form  as  can  be  reliably 
measured  (variable  of  interest  should  always  be  measured  on  a 
continuum,  net  dichotomized) .  Unnecessary  dichotomization 
causes  a  loss  of  power  in  all  analyses.  Consequently,  a  m.uch 
larger  sam.ple  is  necessary  to  achieve  the  same  power. 

Finally,  as  stated  above,  a  more  accurate  estimate  of  the 
Fort  Bragg  Evaluation  Project  effect  size  is  achieved  when 
actual  data  is  utilized  and  a  full  post  hoc  power  analysis  is 
conducted.  The  advisability  of  performing  post  hoc  power 
analysis  is  strongly  supported  by  Cohen  (1988),^^  Rossi 
(1990),-^  Bailar  (1992), and  numerous  authorities  on  power 
analysis  in  the  behavioral /medical  sciences. 
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